Nowadays, Android-based devices such as smart phones, tablets, and recently virtual reality headsets have found increasing usage in our daily lives. Along with the development of software for these devices, new malicious applications are released by intruders, which are more difficult to identify and deal with because they use more sophisticated methods. Although methods have been provided to calculate the security risk and identify malicious apps, but with the expansion of the level and depth of their threats, the need for new methods in this field is still required. In this study, we have presented a new algorithm to calculate the security risk of Android apps, which can be used to identify malicious apps from benign ones. In this algorithm, to estimate the security risk of an input app, the nearest neighbors of the type of malicious apps and the nearest neighbors of the type of normal apps are determined separately using Hamming distance. Then, based on the criteria presented in this article, the security risk of an unknown input app can be computed. After implementing this algorithm and adjusting the parameter of the number of neighbors with the help of real data, extensive various experiments were conducted in order to evaluate the proposed method. In these experiments, the proposed method was compared with three previously known methods in the context of detecting malicious apps, using four different datasets. The results show the higher detection rate of the proposed method in most cases.